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1. INTRODUCTION 

Many methods exist for determining whether someone is lying. Physiological responses such as 
body temperature, blood pressure, heart rate, and respiratory rate, which are detectable by a lie detector, and 
speech, tone, facial expressions, and body language can all serve as a basis for judgment. However, the 
polygraph recording process is an invasive procedure, and its actual application is limited by the consent of 
the person concerned and its immediate effect. When a person lies, the brain sends signals to the facial 
muscles, which makes lying difficult without giving away hints [1], [2]. Therefore, microexpressions have 
been viewed as a possible indicator of lying [3]. Two psychologists, Ekman and Friesen, suggested that 
microexpressions occur when people attempt to conceal their emotions, which is why they proposed the 
facial action coding system (FACS), a system for coding facial actions [1], [4]. 

The majority of microexpressions are characterized by subtle changes in facial features and muscles 
that are rapid, difficult to control, and difficult to feign [5]. Therefore, microexpressions are regarded as 
crucial indicators for lie detection [6]. During an interrogation, the personnel of the Ministry of Justice 
Investigation Bureau typically evaluate the credibility of statements by observing the interrogee’s face and 
body movements. However, the duration of microexpressions ranges from approximately 1/25 to 1/3 s [7], 
and the precise definition of such duration varies [8]. In addition, the variations of microexpressions are 
extremely small and require careful observations to discern [9]. Although training can improve the ability to 
detect microexpressions [10], the recognition success rate is low [7], [8] 

While investigating a crime or collecting evidence, the police may contact a suspect for questioning. 
Nevertheless, the method of questioning used must adhere to the applicable regulations. Illegal and improper 
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interrogation techniques, such as threatening, coercion, deception, and illegal detention, should be avoided 
lest the obtained evidence be rendered useless. The purpose of such interrogation is to coax the interrogee 
into telling the truth and to glean information from their statements to further clarify the case or uncover clues. 
The key to solving many cases lies in the statements of the person being questioned. Therefore, interrogation is 
regarded as one of the most crucial methods used by investigators to discover the facts of a crime. 

If the investigators are unable to confirm the veracity of a statement or determine what the 
interrogee is thinking about, the investigation may be jeopardized or may steer into the wrong direction, 
which not only wastes time and resources but also allows criminals to escape justice. Therefore, under the 
premise of legal law enforcement, investigators must seek effective interrogation techniques. The rest of this 
paper is organized as. Section 2 describes our research methodology, including how we collected the data and 
extracted useful information for further analysis. Section 3 presents our findings and discussion. Finally, 
section 4 shows our conclusions. 


2. METHOD 

The FACS uses 44 action units (AUs) to describe facial expressions [11]. Multiple types of AUs can 
be combined freely, allowing for a variety of expressions. For instance, the combination AU4 (lower 
eyebrow) + AU5 (upper eyelid) + AU24 (lips pressed together) corresponds to the emotional state of anger. 
The facial motion coding system can objectively, precisely, and comprehensively describe facial expressions. 
In the past, the high cost of labeling has severely hampered the progress of AU identification research. This is 
because different AUs are distributed in various regions of the face, expressing subtle changes of different 
intensities. Currently, facial recognition and model learning technologies based on artificial intelligence are 
used to instantly label and publish AU units, allowing for the recognition and evolution of microexpressions. 
Both emotional facial action coding system (EMFACS) [11] and facial action coding system affect 
interpretation dictionary (FACSAID) [12] consider only emotion-related facial actions. Table 1 depicts the 
seven combinations of emotional AUs, and Table 2 depicts the facial muscles responsible for 
microexpressions [13]. 


Table 1. Facial expression calculation from single AUs 


Emotion Action units 
Happiness 6+12 
Sadness 144415 
Surprise 14+2+5B+26 
Fear 1+2+4+5+7+20+26 
Anger 4+5+7+23 
Disgust 9+15+17 
Contempt R12A+R14A 


Table 2. Main action codes 


AU number FACS name Muscular basis 
1 Inner brow raiser Frontalis (pars medialis) 
2 Outer brow raiser Frontalis (pars lateralis) 
4 Brow lowerer Depressor glabellae, depressor supercilii, corrugator supercilii 
5 Upper lid raiser Levator palpebrae superioris, superior tarsal muscle 
6 Cheek raiser Orbicularis oculi (pars orbitalis) 
7 Lid tightener Orbicularis oculi (pars palpebralis) 
9 Nose wrinkler Levator labii superioris alaeque nasi 
10 Upper lip raiser Levator labii superioris, caput infraorbitalis 
11 Nasolabial deepener Zygomaticus minor 
12 Lip corner puller Zygomaticus major 
13 Sharp lip puller Levator anguli oris (also known as caninus) 
14 Dimpler Buccinator 
15 Lip corner depressor Depressor anguli oris (also known as triangularis) 
16 Lower lip depressor Depressor labii inferioris 
17 Chin raiser Mentalis 
18 Lip pucker Incisivii labii superioris and incisivii labii inferioris 
20 Lip stretcher Risorius w/ platysma 
22 Lip funneler Orbicularis oris 
23 Lip tightener Orbicularis oris 
24 Lip pressor Orbicularis oris 
25 Lips part Depressor labii inferioris, or relaxation of mentalis or orbicularis oris 
26 Jaw drop Masseter; relaxed temporalis and internal pterygoid 
27 Mouth stretch Pterygoids, digastric 
28 Lip suck Orbicularis oris 
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In this study, we focused on developing a microexpression recognition system for the personnel of 
the Ministry of Justice Investigation Bureau. The human face produces more than just microexpressions. 
When microexpressions occur, the human face may also change in a variety of ways. Therefore, isolating 
irrelevant information and extracting important information are essential [14], [15]. In addition, compared 
with manual detection, the advantage of using artificial intelligence (AI) to identify microexpressions is that 
as long as the camera objectively captures the identified object, the computer will have obtained the relevant 
data for processing [16]. In other words, by installing cameras, a certain level of recognition can be achieved 
[16], [17]. In this study, we used FaceReader, professional facial expression analysis software, for 
microexpression recognition. 

In addition to the main recognition software, the software used in this study offers an expression 
item comparison analysis module, an expression muscle unit analysis module, and an expression remote 
Photoplethysmography (rPPG) module. The most recent version of the software also incorporates deep neural 
networks, meaning that a deep face model is added to the original active appearance model (AAM) [18]. To 
construct the face module, the AAM thoroughly considers the shape, texture, and local features of the face 
[15]. Subsequently, after face localization, the AAM starts operating and extracts feature data based on 500 
feature points [19]. If the AAM is rendered inoperable because of obstacles such as face occlusion, the deep 
face model continues to function. Currently, this software can recognize eight basic facial expressions, facial 
changes, and physical features. The system also offers a variety of convenient functions such as the ability to 
present recognition results and relevant data in different forms depending on the requirements and to mark 
the selected categories. The open application programming interface facilitates the integration of the system 
data into the recognition system, and the rapid analysis function aids in the sample collection and labeling 
process, which is highly useful in research [20]. 

An AU labeling expert requires at least 30 minutes to analyze a one-minute video of a human face 
[21], [22]. The advantage of machine learning is that it can replace the time-consuming manual labor. The 
emotion recognition processing framework of this system consists of data collection (video), data input, data 
labeling (AUs), machine learning, and recognition output. A flowchart of the proposed system is depicted in 
Figure 1. 
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Figure 1. System flowchart 
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3. RESULTS AND DISCUSSION 

The self-developed system proposed in this study is primarily intended to provide investigation 
bureau personnel with the instant emotional responses of the respondents, thereby enhancing the efficiency of 
case investigations. According to psychologists, when microexpressions occur, facial muscle movements are 
not directly related to individual characteristics, such as gender, age, or ethnicity [23], and the system is 
configured for use in an indoor confined space with no denoising concerns. Thus, during data labeling, only 
facial expressions are considered for emotional analysis. In addition, considering the usage sites, five 
photography conditions are planned to determine the optimal analysis configuration. 


3.1. Real-time facial expressions and action recognition system 

Given the convenience and practicability of actually using the recognition system, we built a 
three-part system consisting of a testing theme documentation system, a dynamic microreaction recording 
system, and a retrospective analysis system. 


3.1.1. Validation of the theme documentation system 

The system can be used to perform theme addition, modification, and deletion as shown in Figure 2. 
The file creation date is automatically generated by the system and cannot be manually entered or altered. 
The ideation and design mode of the questions is based on a single respondent as shown in Figure 3. The user 
can predict the respondent’s emotional response toward a question, and multiple selections can be made. 


Tested Theme File System 


Topic: {f| 1e transection detalis and content of parcel for case no Pi 2020-04-25 | i 


Subject Norman Cheng 
——— 


toie: die [2020080 
sae 


| Create | | Edit | | Delete 


Figure 2. Tested theme file system 


Create Test Questions 
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Theme: Clarify the transaction details and content of 
= parcel for case no. 10311 


Subject: Norman Cheng 


Figure 3. Creating test questions (create/edit) 
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3.1.2. Dynamic microreaction recording system 

When a user accesses the system interface as shown in Figure 4 and clicks the button in the upper- 
right corner, the system starts recording and analyzing emotional responses and body movements. In the 
bottom-right corner of the screen, the user can pause or terminate the recording of the current test item. The 
test time is automatically generated by the system and cannot be manually entered or modified. The system 
logs the interrogation start time and the duration of the current interrogation. The purpose is to remind the 
user to consider the physical and mental effects of interrogation duration on the interrogee. The left-hand side 
records the interrogee’s current emotional response and its intensity as shown in Figure 5. The right-hand 
side displays the interrogee’s predicted emotional response on the basis of predefined questions. When the 
system detects a microexpression, a red frame appears around the label and a warning sound is played. The 
left-hand side indicates the number of times that the microexpression has occurred, and the expression 
indicator light lasts for 3 s. When unexpected expressions appear, a special warning is displayed in the form 
of a blue frame and a unique sound effect. 
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Figure 5. Analysis results of real-time expression reaction intensity 


3.1.3. Review and analysis system 

In Figure 6, The upper-left corner displays the frequency and time points at which the emotion has 
appeared. If the user clicks on the time stamp, the video skips to the corresponding section (5 s of an 
introduction clip). The bottom-left corner displays the current playback speed of the video (x0.5, x1, or x2). 
The right-hand side displays the predicted emotional response of the interrogee on the basis of predefined 
questions. 
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Figure 6. Recording analysis system 


3.2. Analysis of the system configuration conditions 

Regarding the interrogation room, we identified five system configuration factors that may influence 
the microexpression analysis results. These experimental factors included the resolution, shooting angle, 
shooting height, shooting distance, and frame rate (frames per second/FPS) of the camera. The results of the 
experimental tests are as. 


3.2.1. Analysis of the recognition results at different resolutions 

After the captured videos were analyzed using Facereader, editing software was used to adjust the 
video resolution to 600p (800x600) and 480p (640x480). Facereader was then used for analysis and 
comparison as shown in Figure 7. The results indicated that the recognition performance of the 600p videos 
were considerably higher than those of the 480p videos. 


3.2.2. Analysis of the recognition results at different shooting angles 

FaceReader was used to analyze the original videos, and the frontal and side videos (45° and —45°) 
were selected for analysis and comparison. As shown in Figure 8, the recognition results of the three shooting 
angles did not differ considerably. 


w 


m 600P = Original 


N 
u 


= 480P 945° 


n (-45°) - 


N 
N 


BR 


© 
wu 


[=] 

ùn =. 
i 
rasai 
me 


RECOGNITION RESULT 
r A 
RECOGNITION RESULT 


o 
-mm 
m 
o 
R 
js -- = 
w j EEOC EL LEE LEELA 
> Wi MMMM 
E 
= WP, 


AU06 AU012 TRIAL 
TRIAL 


Figure 7. Recognition results of AU6 and AU12 Figure 8. Recognition results of AU6 and AU12 
at different resolutions at different shooting angles 
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3.2.3. Analysis of the recognition results at different shooting heights 

FaceReader was used to analyze the original video, and the frontal and elevation videos (45°) were 
selected for comparison. As shown in Figure 9, the recognition results of the two shooting heights did not 
differ considerably. 


3.2.4. Analysis of the recognition results at different shooting distances 

The video was played back on a laptop, and the camera was positioned 30, 100, 200, and 300 cm 
away from the laptop for analysis. As shown in Figure 10, the recognition results obtained at 30, 100, and 
200 cm did not differ considerably. Nonetheless, at a distance of 300 cm, the large gap hindered the 
recognition of microexpressions. 
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Figure 9. Recognition results of AU6 and AU12 Figure 10. Recognition results of AU6 and AU12 
at different shooting heights at different shooting distances 


3.2.5. Analysis of the recognition results at different frame rates 

As shown in Figure 11, the recognition capacity of the high-speed camera (190 FPS) in recognizing 
happy emotions (AU6+AU12) was higher than that of the conventional camera (30 FPS). This is because 
FaceReader captured clearer microexpressions during video output. 
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Figure 11. Recognition results of AU6 and AU12 at different frame rates 


4. CONCLUSION 
In today’s world of rapid technological advancement, an increasing number of criminal cases are 
being investigated with the aid of scientific methods as auxiliary tools. In this study, we proposed a real-time 
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microexpression recognition system for a specific purpose. This system employs FaceReader to gather 
information for real-time analysis and aid law enforcement officers in their investigations. To ensure the 
accuracy of analysis, the system was evaluated under various conditions. However, the duration of 
microexpressions is too short, and the intensity of changes is low. When these characteristics are combined 
with the complexity of human emotions, this leaves large room for exploration in the presentation of 
microexpressions. Currently, the number of videos available for training and analysis is relatively small, and 
the uneven distribution of video lengths, sample frame rates, and sample types affects the recognition of 
microexpressions [23]. Therefore, the system must be continuously updated in the future. 
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